Evaluating DUC 2005 using Basic Elements
نویسندگان
چکیده
In this paper we introduce Basic Elements, a new way of automating the evaluation of text summaries. We show that this method correlates better with human judgments than any other automated procedure to date, and overcomes the subjectivity/variability problems of manual methods that require humans to preprocess summaries to be evaluated. This is demonstrated on DUC 2005 peer systems and peer-produced summaries.
منابع مشابه
Automated Summarization Evaluation with Basic Elements
As part of evaluating a summary automatically, it is usual to determine how much of the contents of one or more human-produced ‘ideal’ summaries it contains. Past automated methods such as ROUGE compare using fixed word ngrams, which are not ideal for a variety of reasons. In this paper we describe a framework in which summary evaluation measures can be instantiated and compared, and we impleme...
متن کاملERSS 2005: Coreference-Based Summarization Reloaded
We present ERSS 2005, our entry to this year’s DUC competition. With only slight modifications from last year’s version to accommodate the more complex context information present in DUC 2005, we achieved a similar performance to last year’s entry, ranking roughly in the upper third when examining the ROUGE-1 and Basic Element score. We also participated in the additional manual evaluation base...
متن کاملEvaluating DUC 2004 Tasks with the QARLA Framework
This papers reports the application of the QARLA evaluation framework to the DUC 2004 testbed (tasks 2 and 5). Our experiment addresses two issues: how well QARLA evaluation measures correlate with human judgements, and what additional insights can be provided by the QARLA framework to the DUC evaluation exercises.
متن کاملSummarization Evaluation Using Transformed Basic Elements
This paper describes BEwTE (Basic El ements with Transformations for Evalua tion), an automatic system for summa rization evaluation. BEwTE is a new, more sophisticated implementation of the BE framework that uses transformations to match BEs (minimallength syntacti cally wellformed units) that are lexically different yet semantically similar. We demonst...
متن کاملAutomatic Evaluation of an Abscissa of Convergence for Inverse Laplace Transform
An automatic algorithm evaluating numerically an abscissa of convergence of the inverse Laplace transform is introduced.
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2005